Skip to content

ADR-0001: Scope compliance claims to implemented controls - #1

Merged
nicholas-ruest merged 3 commits into
mainfrom
adr/0001-compliance-claims-remediation
Jul 27, 2026
Merged

ADR-0001: Scope compliance claims to implemented controls#1
nicholas-ruest merged 3 commits into
mainfrom
adr/0001-compliance-claims-remediation

Conversation

@nicholas-ruest

Copy link
Copy Markdown
Member

Implements docs/adr/ADR-0001-compliance-claims-remediation.md (Phase 1 — documentation-only). The ADR itself is included in this PR since it was not previously committed.

Why

README.md presented SOC 2 Type II, HIPAA, and data residency as delivered capabilities. They are not. There is no audit artifact, and the compliance service backing those claims has never successfully served a request — all 15 tables it queries are undefined in the only migration that exists. Representing unaudited software as SOC 2 Type II certified is a misrepresentation independent of code quality, and HIPAA language specifically invites healthcare customers to route PHI into a system whose PHI logging currently throws on every call.

The engineering work behind these claims is real and competent; the README overstates its status. This PR corrects the status only. No services/compliance/ code is touched — the technical fix is ADR-0002.

What changed

README.md

  • :33-36 — section retitled "Compliance & Governance (Design Stage)". SOC 2 reworded to "SOC 2 control mapping and evidence-collection design (no audit performed)"; HIPAA to "HIPAA-oriented control catalog covering §164.308/310/312 (design stage, not operational)"; Data Residency to "Data residency policy model (design stage)".
  • :208 — compliance service description reworded to design-stage, marked not deployed, linked to the ADR.
  • :210-216 — added a Status column, every row Design.
  • "BAA management" deleted from both :35 and :214, not reworded. A Business Associate Agreement is a signed contract under §164.502(e), not a software feature; to a healthcare buyer "BAA management" reads as "Agentics will sign a BAA."
  • :218 — note above the endpoint list stating the service is deployed in no environment.
  • :470 — Phase 5 row split into "AI/ML Platform — Complete" and "Compliance — Design Complete, Implementation In Progress".
  • :473-480 — section retitled "Security"; SOC 2 Type II, HIPAA, and GDPR bullets deleted. Encryption retained unchanged — it is genuinely implemented — now citing deployment/terraform/main.tf:243,298-299,340-344. I verified all five of those line citations resolve to real storage_encrypted, at_rest_encryption_enabled, transit_encryption_enabled, and SSE-KMS settings before asserting them.

deployment/compliance/soc2-evidence-collection.yaml — header marked Status: Design Specification -- no audit performed; the 10 unresolvable path: entries marked # TODO: artifact does not yet exist. I resolved all 14 paths mechanically; exactly the 10 named in the ADR are missing and the other 4 resolve, confirming Finding 7.

New: scripts/check_compliance_claims.py — ADR Verification 2. Greps README.md and docs/ for SOC 2|SOC2|HIPAA compliant|BAA|GDPR compliant|certified and fails unless each hit is within 3 lines of a resolvable repo path or a design-stage qualifier.

New: scripts/check_evidence_paths.py — ADR Verification 3. Resolves every path: in the evidence manifest, fails on any that neither exists nor is marked # TODO.

Test output — real, not fabricated

Both checks pass on this branch, and both were verified to actually fail when a regression is introduced (not just to pass trivially):

$ python3 scripts/check_compliance_claims.py
check_compliance_claims: OK -- no new unbacked claims (14 baselined).

$ python3 scripts/check_evidence_paths.py
check_evidence_paths: OK -- 14 path entries, all resolvable or marked TODO.

=== NEGATIVE TEST: append "We are SOC 2 Type II certified." to README ===
New unbacked compliance claims (need a resolvable link or a 'design stage'
qualifier within 3 lines) -- see docs/adr/ADR-0001-compliance-claims-remediation.md:

  README.md:515: - We are SOC 2 Type II certified.
exit=1 (expect 1)

=== NEGATIVE TEST: point an evidence path at a nonexistent directory ===
Evidence paths that do not resolve and are not marked '# TODO':

  soc2-evidence-collection.yaml:231: deployment/nonexistent/
exit=1 (expect 1)

soc2-evidence-collection.yaml and .github/workflows/ci.yml were both re-parsed with a YAML parser after editing to confirm they remain valid.

Deviations from the ADR — please review these three

1. The claims check ships with a 14-entry baseline, and I did not edit the docs it flags.

ADR Verification 2 specifies the check covers README.md and docs/. Running it repo-wide surfaced 14 pre-existing unbacked claims outside the ADR's enumerated edit list, six of which are substantively worse than anything in the README:

Location Text
docs/specification.md:776 soc2: Type II certified
docs/specification-summary.md:119 SOC 2 Type II: Certified
docs/architecture/DELIVERABLES.md:168 SOC 2 Type II: Certified
docs/architecture/DELIVERABLES.md:668 SOC 2 Type II certification
docs/specification.md:1070 SOC 2 Type II compliance
docs/architecture/DATA_STORAGE_DELIVERABLES.md:495 Compliance: GDPR, SOC 2, HIPAA compatible

I had three options and took the third: rewrite six documents the ADR did not scope; weaken the check until it passes; or enable the check as specified with an explicit in-tree baseline (docs/compliance-claims-baseline.txt). I did not rewrite them because these read as requirement statements in specification documents — the surrounding lines say "Full compliance required", "Mitigation required" — and ADR-0001 Finding 9 shows the ADR author reviewed doc status deliberately. Reclassifying six design docs is a judgment call that belongs to whoever owns them, not to this PR.

The baseline is matched on file + exact line text, not line number, so editing any baselined line re-surfaces it as new and it cannot silently absorb a fresh claim. It is a visible backlog, not a hidden exemption; the header says so. If you would rather I fix those six lines now, say so and I will. I flag specification.md:776 in particular: soc2: Type II certified sitting next to hipaa: Not supported (medical data prohibited) is a directly contradictory pair worth a human decision.

2. The endpoint line for BAA was reworded, not deleted. The ADR says delete "BAA management" from :35 and :214 (done, both deleted). :223 is an endpoint listing for a route that genuinely exists, so deleting it would make the API docs wrong. I reworded it to "BAA registry records, design stage (tracking only; does not execute agreements)" — the registry/execution distinction the ADR's Decision item 3 draws. Say the word if you want it gone entirely.

3. CI wiring is NOT in this PR and must be applied by hand. I wrote and validated the workflow job, but the push was rejected: refusing to allow a Personal Access Token to create or update workflow .github/workflows/ci.yml without workflow scope. The two checks therefore do not run in CI until someone with workflow scope adds this. Add to .github/workflows/ci.yml:

  # ADR-0001: compliance claims must be backed by an artifact
  compliance-claims:
    name: Compliance Claims Check
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Check compliance claims are backed
        run: python3 scripts/check_compliance_claims.py

      - name: Check SOC 2 evidence paths resolve
        run: python3 scripts/check_evidence_paths.py

and add compliance-claims to the ci-success job's needs: list and its result check. (Verified locally: the edited ci.yml parses and yields 12 jobs.)

Migration execution status

Not applicable — this PR contains no migration and no code change. Nothing was executed against any database. The schema work is ADR-0002.

Left undone (by design — ADR Phases 2-4)

Everything in ADR-0001 Phases 2, 3, and 4 is out of scope here; Phase 1 was specified to ship independently. Phases 2-3 (the 15-table migration, auth on the compliance routers, deterministic pseudonymize(), fail-closed key loading, enforced PHI logging) are ADR-0002, in a separate PR. Phase 4 — policy artifacts, a third-party §164.308(a)(1)(ii)(A) risk analysis, counsel-produced BAA template, and an independent CPA firm's Type II engagement — is not engineering work and cannot be done in a PR.

Per ADR Verification 6, SOC 2 Type II language may return to README.md only alongside an auditor's report reference, and HIPAA/BAA language only alongside the risk analysis and BAA template. This PR does not unblock either.

Removes the SOC 2 Type II, HIPAA, and GDPR assertions from README.md and
replaces them with design-stage language, since no audit artifact exists and
the compliance service has never run against a real schema. Deletes "BAA
management" outright: a Business Associate Agreement is a signed legal
contract, not a software feature. Retains the encryption claim, which is
genuinely implemented, with a pointer to the Terraform resources.

Marks soc2-evidence-collection.yaml as a design specification and flags its
10 unresolvable evidence paths with TODO.

Adds two CI checks so this cannot regress: check_compliance_claims.py fails
on a compliance claim not backed by a resolvable link or a design-stage
qualifier, and check_evidence_paths.py fails on an unmarked evidence path
that does not exist.

Implements copilot-agent/docs/adr/ADR-0001-compliance-claims-remediation.md
Adds the compliance-claims job the ADR's Verification section calls for. It was
written and validated with the original change but could not be pushed then: the
token lacked workflow scope.

Uses needs['compliance-claims'].result rather than dot access in the ci-success
result check -- a hyphenated job id is parsed as subtraction in a GitHub
expression, so the dot form would not have evaluated correctly.

Implements copilot-agent/docs/adr/ADR-0001-compliance-claims-remediation.md
Removes compliance-claims from the ci-success needs list. That umbrella job is
currently red on main for unrelated reasons -- format, lint, security, test and
coverage all fail there -- so nesting a passing documentation check inside it
masked a meaningful signal behind debt the check cannot influence.

The job still runs on every push and pull request and reports its own status.
If it should block merges, make it a required check in branch protection, which
gates on it directly rather than through an aggregate.

This also makes the ci.yml change purely additive: the ci-success block is now
byte-identical to main.

Implements copilot-agent/docs/adr/ADR-0001-compliance-claims-remediation.md
@nicholas-ruest
nicholas-ruest merged commit c2cfbaf into main Jul 27, 2026
5 of 21 checks passed
@nicholas-ruest
nicholas-ruest deleted the adr/0001-compliance-claims-remediation branch July 27, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant